Mistral-7B-v0.1 is a pre-trained generative large language model with 7 billion parameters, which outperforms the Llama 2 13B model in multiple benchmark tests. It adopts an advanced Transformer architecture design, including Grouped Query Attention and Sliding Window Attention mechanisms.

Natural Language ProcessingPytorchEnglish

Hunyuan-7B is an open-source large language model released by Tencent. It has the ability to process 256K long texts and uses the Grouped Query Attention (GQA) mechanism, performing excellently among Chinese 7B dense models.

Natural Language ProcessingTransformersEnglish

MobileLLM is an autoregressive language model optimized by Meta specifically for resource-constrained device-side applications. It adopts an optimized Transformer architecture and integrates key technologies such as the SwiGLU activation function, a deep and narrow architecture, embedding sharing, and grouped query attention. It performs excellently in zero-shot commonsense reasoning tasks.

Natural Language ProcessingTransformers

Fox-1-1.6B is a small language model with 1.6 billion parameters developed by TensorOpera AI, which adopts the Grouped Query Attention mechanism and is trained on 3 trillion text and code data.

Natural Language ProcessingTransformersEnglish

smol_llama is a small decoder model with 220M parameters, equipped with the GQA (Grouped Query Attention) mechanism, suitable for tasks such as text generation.

Natural Language ProcessingTransformersEnglish

StripedHyena-Nous-7B is a hybrid-architecture chat model combining multi-head grouped query attention and gated convolutions, supporting long-context processing.

Natural Language ProcessingTransformersEnglish

smol_llama-101M-GQA is a small decoder model with a total of 101M parameters. It adopts the GQA (Grouped Query Attention) architecture, having 6 hidden layers and a hidden layer size of 768. This model was pre-trained on a single GPU for 5 compute days, supports a context length of 1024, and focuses on providing efficient text generation capabilities.

Natural Language ProcessingTransformersEnglish

DeciLM 6B is a 5.7-billion-parameter decoder-only text generation model developed by Deci. It uses variable grouped query attention technology to achieve the best balance between performance and computational efficiency. This model has been instruction fine-tuned on the OpenOrca dataset, supports a context window of 4096 tokens, and can achieve a throughput up to 15 times that of Llama 2 7B.

Natural Language ProcessingTransformersEnglish